187 research outputs found
Methods combination and ML-based re-ranking of multiple hypothesis for question-answering systems
International audienceQuestion answering systems answer correctly to different questions because they are based on different strategies. In order to increase the number of questions which can be answered by a single process, we propose solutions to combine two question answering systems, QAVAL and RITEL. QAVAL proceeds by selecting short passages, annotates them by question terms, and then extracts from them answers which are ordered by a machine learning validation process. RITEL develops a multi-level analysis of questions and documents. Answers are extracted and ordered according to two strategies: by exploiting the redundancy of candidates and a Bayesian model. In order to merge the system results, we developed different methods either by merging passages before answer ordering, or by merging end-results. The fusion of end-results is realized by voting, merging, and by a machine learning process on answer characteristics, which lead to an improvement of the best system results of 19 %
On the Usability of Transformers-based models for a French Question-Answering task
For many tasks, state-of-the-art results have been achieved with
Transformer-based architectures, resulting in a paradigmatic shift in practices
from the use of task-specific architectures to the fine-tuning of pre-trained
language models. The ongoing trend consists in training models with an
ever-increasing amount of data and parameters, which requires considerable
resources. It leads to a strong search to improve resource efficiency based on
algorithmic and hardware improvements evaluated only for English. This raises
questions about their usability when applied to small-scale learning problems,
for which a limited amount of training data is available, especially for
under-resourced languages tasks. The lack of appropriately sized corpora is a
hindrance to applying data-driven and transfer learning-based approaches with
strong instability cases. In this paper, we establish a state-of-the-art of the
efforts dedicated to the usability of Transformer-based models and propose to
evaluate these improvements on the question-answering performances of French
language which have few resources. We address the instability relating to data
scarcity by investigating various training strategies with data augmentation,
hyperparameters optimization and cross-lingual transfer. We also introduce a
new compact model for French FrALBERT which proves to be competitive in
low-resource settings.Comment: French compact model paper: FrALBERT, Accepted to RANLP 202
Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis
International audienceManual corpus annotation is getting widely used in Natural Language Processing (NLP). While being recognized as a difficult task, no in-depth analysis of its complexity has been performed yet. We provide in this article a grid of analysis of the different complexity dimensions of an annotation task, which helps estimating beforehand the difficulties and cost of annotation campaigns. We observe the applicability of this grid on existing annotation campaigns and detail its application on a real-world example
On the cross-lingual transferability of multilingual prototypical models across NLU tasks
Supervised deep learning-based approaches have been applied to task-oriented
dialog and have proven to be effective for limited domain and language
applications when a sufficient number of training examples are available. In
practice, these approaches suffer from the drawbacks of domain-driven design
and under-resourced languages. Domain and language models are supposed to grow
and change as the problem space evolves. On one hand, research on transfer
learning has demonstrated the cross-lingual ability of multilingual
Transformers-based models to learn semantically rich representations. On the
other, in addition to the above approaches, meta-learning have enabled the
development of task and language learning algorithms capable of far
generalization. Through this context, this article proposes to investigate the
cross-lingual transferability of using synergistically few-shot learning with
prototypical neural networks and multilingual Transformers-based models.
Experiments in natural language understanding tasks on MultiATIS++ corpus shows
that our approach substantially improves the observed transfer learning
performances between the low and the high resource languages. More generally
our approach confirms that the meaningful latent space learned in a given
language can be can be generalized to unseen and under-resourced ones using
meta-learning.Comment: Accepted to the ACL workshop METANLP 202
Survey on Evaluation Methods for Dialogue Systems
In this paper we survey the methods and concepts developed for the evaluation
of dialogue systems. Evaluation is a crucial part during the development
process. Often, dialogue systems are evaluated by means of human evaluations
and questionnaires. However, this tends to be very cost and time intensive.
Thus, much work has been put into finding methods, which allow to reduce the
involvement of human labour. In this survey, we present the main concepts and
methods. For this, we differentiate between the various classes of dialogue
systems (task-oriented dialogue systems, conversational dialogue systems, and
question-answering dialogue systems). We cover each class by introducing the
main technologies developed for the dialogue systems and then by presenting the
evaluation methods regarding this class
Benchmarking Transformers-based models on French Spoken Language Understanding tasks
In the last five years, the rise of the self-attentional Transformer-based
architectures led to state-of-the-art performances over many natural language
tasks. Although these approaches are increasingly popular, they require large
amounts of data and computational resources. There is still a substantial need
for benchmarking methodologies ever upwards on under-resourced languages in
data-scarce application conditions. Most pre-trained language models were
massively studied using the English language and only a few of them were
evaluated on French. In this paper, we propose a unified benchmark, focused on
evaluating models quality and their ecological impact on two well-known French
spoken language understanding tasks. Especially we benchmark thirteen
well-established Transformer-based models on the two available spoken language
understanding tasks for French: MEDIA and ATIS-FR. Within this framework, we
show that compact models can reach comparable results to bigger ones while
their ecological impact is considerably lower. However, this assumption is
nuanced and depends on the considered compression method.Comment: Accepted paper at INTERSPEECH 202
Lifelong learning and task-oriented dialogue system: what does it mean?
International audienceThe main objective of this paper is to propose a functional definition of lifelong learning system adapted to the framework of task-oriented system. We mainly identified two aspects where a lifelong learning technology could be applied in such system: improve the natural language understanding module and enrich the database used by the system. Given our definition, we present an example of how it could be implemented in an actual task-oriented dialogue system that is developed in the LIHLITH project
- …